Welcome to Building Event
Streaming Pipelines Using Kafka. After watching this video, you will be able to: Describe the core components of Kafka. Use Kafka to publish (write) and
subscribe to (read) streams of events. Use Kafka to consume events, either
as they occur or retrospectively, and Describe an end-to-end event
streaming pipeline example. A Kafka cluster contains one or many
brokers. You may think of a Kafka broker as a dedicated server to receive,
store, process, and distribute events. Brokers are synchronized and managed by
another dedicated server called ZooKeeper. For example, here we have A log topic and a transaction topic in broker 0. A payment topic and a GPS topic in broker 1. and, a user click topic and user
search topic in broker 2. Each broker contains one or many topics. You
can think of a topic as a database to store specific types of events, such as logs,
transactions, and metrics, for example. Brokers manage to save published
events into topics and distribute the events to subscribed consumers. Like many other distribution systems, Kafka implements the concepts of
partitioning and replicating. It uses topic partitions and replications
to increase fault-tolerance and throughput so that event publication and consumption can
be done in parallel with multiple brokers. In addition, even if some brokers
are down, Kafka clients are still able to work with the target topics
replicated in other working brokers. For example: A log topic has been separated
into two partitions: 0, 1, and a user topic has been separated
into two partitions: 0, 1. And each topic partition is duplicated into two
replications and stored in different brokers. The Kafka CLI, or command line interface
client provides a collection of powerful script files for users to build
an event streaming pipeline: The Kafka-topics script is the
one you probably will be using often to manage topics in a Kafka
cluster. It is straightforward. Let’s have a look at some common usages: This first one is to create a topic. Here we are trying to create
a topic called ‘log_topic’ with two partitions and two replications. One important note here is that many Kafka
commands, like kaf-topics, require users to refer to a running Kafka cluster with a host and
a port, such as a localhost with the port 9092. After you have created some topics, you can check all created topics in
the cluster using the ‘list option.’ And, if you want to check more details of a topic, such as partitions and replications,
you can use the ‘describe option’. And you can delete a topic
using the ‘delete option.’ Next, you will find out more about
publishing events using Kafka producers. Features of Kafka producer: They are client applications that
publish events to topic partitions according to the same order as they are published. When publishing an event in Kafka producer, an
event can be optionally associated with a key. Events associated with the same key will
be published to the same topic partition. Events not associated with any key will be
published to topic partitions in rotation. Let’s see how you can publish events to topic
partitions using the following example: Suppose you have an event source 1,
which generates various log entries, and an event source 2, which generates
user-activity tracking records. Then, you can create a Kafka producer to
publish log records to log topic partitions, and a user producer to publish user-activity
events to user topic partitions, respectively. When you publish events in producers, you
can choose to associate events with a key, for example, an application name or a user ID. Similar to the Kafka topic CLI, Kafka provides the Kafka producer
CLI for users to manage producers. The most important aspect is starting a
producer to write or publish events to a topic: Here you start a producer and point it
to the log_topic, then you can type some messages in the console to start publishing
events. For example, log1, log2, and log3. You can provide keys to events to make sure the events with the same key
will go to the same partition. Here you are starting a producer to user_topic,
with the parse.key option to be true, and you also specify the
key.separator to be comma. Then you can write messages as follows:
key: ‘user1’, value: ‘login website’. Key: ‘user1’, value: ‘click the top item’. And, key: ‘user1’, value: ‘logout
website’. Accordingly, all events about user1 will be saved in the same partition
to facilitate the reading for consumers. Once events are published and
properly stored in topic partitions, you can create consumers to read them. Consumers are client applications that
can subscribe to topics and read the stored events. Then event destinations can
further read events from Kafka consumers. Consumers read data from topic partitions
in the same order as they are published. Consumers also store an offset for each
topic partition as the last read position. With the offset, consumers are
guaranteed to read events as they occur. A playback is also possible by
resetting the offset to zero. This way the consumer can read all events in
the topic partition from the beginning. In Kafka, producers and
consumers are fully decoupled. As such, producers don’t need to synchronize with
consumers, and after events are stored in topics, consumers can have independent
schedules to consume them. To read published log and
user events from topic partitions, you will need to create
log and user consumers, and make them subscribe to corresponding topics. Then Kafka will push the events
to those subscribed consumers. Then, the consumers will further
send to event destinations. To start a consumer is also easy,
using the Kafka consumer script. Let’s read events from the log_topic. You just need to run the
Kafka-console-consumer script and specify a Kafka cluster
and the topic to subscribe to. Here, you can subscribe to and read
events from the topic log_topic. Then the started consumer will read only the new
events, starting from the last partition offset. After those events are consumed, the
partition offset for the consumer will also be updated and committed back to Kafka. Very often a user wants to read
all events from the beginning, as a playback of all historical events. To do so, you just need to add
the ‘from beginning’ option. Now, you can read all events
starting from offset 0. Let’s have a look at a more
concrete example to help you understand how to build an event
streaming pipeline end-to-end. Suppose you want to collect and analyze weather
and Twitter event streams, so that you can correlate how people talk about extreme weather
on Twitter. Here you can use two event sources: IBM Weather API to obtain real-time and
forecasted weather data in JSON format. And Twitter API to obtain real-time tweets
and mentions, also in JSON format. To receive weather and twitter JSON data
in Kafka, you then create a weather topic and a Twitter topic in a Kafka cluster,
with some partitions and replications. To publish weather and Twitter
JSON data to the two topics, you need to create a weather
producer and a Twitter producer. The event’s JSON data will be serialized
into bytes and saved in Kafka topics. To read events from the two topics,
you need to create a weather consumer and a Twitter consumer. The bytes stored in Kafka
topics will be deserialized into event JSON data. If you now want to transport the weather and Twitter event JSON data from the
consumers to a relational database, you will use a DB writer to parse those
JSON files and create database records. And then you can write those records into
a database using SQL insert statements. Finally, you can query the database
records from the relational database and visualize and analyze them in a dashboard
to complete the end-to-end pipeline. In this video, you learned that: The core components of Kafka are: Brokers: The dedicated server to receive,
store, process, and distribute events. Topics: The containers or databases of events. Partitions: Divide topics into different brokers. Replications: Duplicate
partitions into different brokers. Producers: Kafka client applications
to publish events into topics. And, consumers: Kafka client
applications subscribed to topics and read events from them. You also learned the Kafka-topics CLI manages topics. The Kafka-console-producer CLI manages producers. And finally, the Kafka-console-consumer
manages consumers.